Answer: Sometimes it's a matter of opinion when or whether a piece has modulated to a new key, or is just tossing in a foreign chord for a moment. The usual principle, though, is that to be a real modulation to a new key you must be seeing at least one V-I cadence in the new key. If that cadence is not followed by more typical use of the new key then it's not really a true modulation, but just an exotic moment.

If a single E major chord is all you have then this doesn't sound like a change of key. The E could be analyzed in G major as VI - an altered form of the normal vi chord.

What chord follows the E? If it's the ii chord (that is, A minor), then the E would be analyzed as V of ii (a secondary dominant). What counts is the function of that E. I'd expect that raised G in the E chord to be going somewhere; the next chord probably includes an A.

If you can tell me more about the piece, or email an image, I could provide a more complete answer to this question.

[Writer hasn't answered that followup question, so we'll have to leave it at that!].